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Few elementary students on the Commonwealth of the Northern Mariana Islands (CNMI) are scoring at 


grade level or higher on the ACT Aspire reading assessment. To better understand factors associated with 
the reading proficiency of CNMI grade 3 students, stakeholders there asked the Regional Educational 
Laboratory Pacific to examine the demographic characteristics and education experiences of students 
who demonstrated reading proficiency by grade 3. The study focused on grade 3 students who were 
enrolled in CNMI public schools from 2014/15 to 2018/19. It found that female students, students who 
did not receive free or reduced-priced lunch, students who were older at the time of kindergarten entry, 
Filipino students, and students who did not change schools were more likely to demonstrate reading 
proficiency in grade 3 than other students. There was no difference in grade 3 reading proficiency between 
students who had enrolled in Head Start and students who had not. 


Few elementary students in the Commonwealth of the Northern Mariana Islands (CNMI) are reading at grade 
level or higher, according to standardized assessments. In 2015, 17 percent of grade 3 students in the CNMI Public 
School System scored at or above proficiency on the ACT Aspire Reading assessment (Commonwealth of the 
Northern Mariana Islands Office of Accountability, Research, and Evaluation, 2015). Concerned by this low per- 
centage, CNMI Public School System leadership prioritized reading achievement across its elementary schools 
by investing in teacher professional development on instruction for foundational reading skills and by hiring lit- 
eracy coaches to provide continuous supports for teachers’ reading instruction (Commonwealth of the Northern 
Mariana Islands Office of Accountability, Research, and Evaluation, 2015). Although the percentage of students 
meeting reading proficiency benchmarks rose to 24 percent in 2019, school proficiency rates varied widely—from 
10 percent to 46 percent (Commonwealth of the Northern Mariana Islands Office of Accountability, Research, and 
Evaluation, 2019a). 


Prior research indicates that students who meet reading benchmarks in grade 3 are likely to meet benchmarks 
across middle and high school (Hein et al., 2013; Singh, 2013). Meeting reading benchmarks is also positively cor- 
related with other key academic skills (such as math skills development; see Gersten et al., 2005) and social skills 
(Miles & Stipek, 2006) in later elementary grades. Moreover, research suggests that reading proficiency positively 
influences postsecondary enrollment. Students who meet grade 3 reading proficiency benchmarks are more 
likely to graduate from high school and enroll in college than those who do not 
meet those benchmarks in grade 3 (Lesnick et al., 2010). For additional 
information, including 

Though evidence points to skills such as oral reading fluency and vocabulary background on the study, 
as strong predictors of grade 3 reading proficiency (Lonigan et al., 2018; Quinn technical methods, 

et al., 2015), studies also suggest associations between students’ demographic 
characteristics and education experiences and their reading proficiency (Canto 
& Proctor, 2013; Mulligan et al., 2018; Scammacca et al., 2020). For example, 
mobility (the number of times a student changes schools) can negatively impact 
reading and other academic outcomes (Ashby, 2010). Growing evidence suggests 
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a positive relationship between enrollment in a high-quality early childhood education program, such as Head 
Start, and the development of skills and abilities that support students’ short- and long-term success, including 
reading outcomes (Meloy et al., 2019; Wechsler et al., 2016; Yoshikawa et al., 2016). For instance, the federally 
funded Head Start Impact Study, a randomized controlled trial using a nationally representative sample of stu- 
dents, found that Head Start enrollment had significant positive effects on the development of children’s literacy 
skills in kindergarten (Puma et al., 2010). 


Families in the CNMI have access to early childhood education services, but data suggest that enrollment rates 
are low. Families can choose early childhood education services provided by the CNMI Public School System, 
which operates the Head Start program, or services provided through 25 licensed facilities and nine license- 
exempt home care providers (Commonwealth of the Northern Mariana Islands Preschool Development Grant 
Birth through Five, 2020). Head Start is the largest provider of early childhood education in the CNMI, with nine 
Head Start centers across the islands of Saipan, Tinian, and Rota, as of 2019 (Commonwealth of the Northern 
Mariana Islands Office of Accountability, Research, and Evaluation, 2019b). About 44 percent of families with 
young children in the CNMI have a household income below the federal poverty level, suggesting that their chil- 
dren are eligible to enroll in Head Start (Commonwealth of the Northern Mariana Islands Department of Com- 
merce, 2018). In 2018, however, the total Head Start enrollment count was 393 students, which is less than half 
of CNMI’s eligible three- and four-year-old children (Commonwealth of the Northern Mariana Islands Office of 
Accountability, Research, and Evaluation, 2019b). Although this is slightly lower than the 2021 funded enrollment 
of 396 students, the total number of students enrolled in Head Start has declined since 2015, when funded enroll- 
ment was 462 students (Commonwealth of the Northern Mariana Islands Office of Accountability, Research, and 
Evaluation, 2019b). CNMI stakeholders have suggested that enrollment in an early childhood education program 
like Head Start could be associated with grade 3 reading proficiency. Therefore, these low enrollment rates are a 
cause for concern. 


The CNMI Public School System supports students not only in meeting academic achievement standards but also 
in acquiring “essential skills for lifelong learning, and ha[ving] the ability to transition into higher education and/ 
or the workforce” (Commonwealth of the Northern Mariana Islands Public School System, 2021, para. 3). There- 
fore, understanding the relationships between its grade 3 students’ demographic characteristics and education 
experiences and their reading performance is of critical interest. CNMI Public School System stakeholders sought 
support from the Regional Educational Laboratory Pacific to investigate these relationships in order to inform 
their efforts to improve literacy and better prepare students for academic and life success. 


Research question 

The study focused on students who enrolled in grade 3 in a CNMI public school from 2014/15 to 2018/19. It exam- 
ined the association between those students’ demographic characteristics and education experiences and their 
grade 3 reading proficiency. The demographic characteristics included gender, free or reduced-price lunch receipt 
(a proxy for family income), age at kindergarten entry, and ethnicity. The education experiences included Head 
Start enrollment and mobility. 


The following research question guided the study: 


e Which student demographic characteristics and education experiences were associated with grade 3 reading 
proficiency? 


Key terms are defined in box 1. Data sources, the study sample, and analytic methods are summarized in box 2 
and detailed in appendix B. 


REL 2021-116 2 


Box 1. Key terms 


Age at kindergarten entry. A student’s age at kindergarten entry, ranging from 56 months to 72 months. According to CNMI 
Public School System policy, a child must be 5 years old on or before September 30 of the school year to enter kindergarten (Com- 
monwealth Law Revision Commission, 2019).' 


Commonwealth of the Northern Mariana Islands (CNMI) Public School System. The public school system of CNMI that over- 
sees all public K-12 schools on the islands of CNMI: Saipan, Tinian, and Rota. Since 1984 the CNMI Public School System has served 
as the agency operating the CNMI Head Start program. 


Education experience. Formal and informal instruction, interactions, programs, and other experiences that students could 
undergo during their schooling. Education experiences in this study included Head Start enrollment and mobility. 


Ethnicity. The reported ethnicity of a student. Three ethnicity categories were reported: Chamorro or Carolinian,? Filipino, and 
other (which combines all remaining ethnic categories). 


Free or reduced-price lunch receipt. Whether a student received free or reduced-price lunch at kindergarten entry (a proxy for 
family income). The National School Lunch Program provides free or reduced-cost meals to eligible students during the school 
day. Students are eligible if their household income is at or below 185 percent of the federal income poverty level and their 
parents or other caregivers enroll them in the program.? 


Gender. The reported gender of a student. Two gender categories were reported: female or male. 


Grade 3 reading proficiency. Whether a student scored at the Ready or Exceeding benchmark on the ACT Aspire reading assess- 
ment at the end of grade 3, demonstrating reading proficiency. The ACT Aspire reading assessment is a computer-based, stan- 
dardized benchmark assessment of reading skills administered in the spring to all CNMI grade 3 students. It covers informational 
and literacy text skills related to identifying key ideas and details in text, analyzing text structure and purpose, and integrating 
knowledge and ideas from text. Data were provided as a categorical variable indicating whether a student scored significantly 
below (Needs Support), below (Close), at (Ready), or above (Exceeding) grade-level benchmarks for proficiency. The study team 
then converted that categorical variable to a binary variable indicating whether the student was proficient (received a benchmark 
score of Ready or Exceeding) or not proficient (received a benchmark score of Close or Needs Support) in reading in grade 3. 


Head Start enrollment. Whether a student had enrolled in Head Start for at least one year. 


Mobility. In this study the number of schools a student attended between kindergarten and grade 3. Two mobility categories 
were used: attended one school and attended more than one school. An insufficient number of students attended three or four 
schools to examine any of those groups of students separately. 


Predicted probability. The predicted probability (in percent) of an outcome occurring for a given value of a predictor, with all 
other predictors held constant at their mean. This study reports the probability of a student demonstrating proficiency (receiving 
a benchmark score of Ready or Exceeding) rather than not demonstrating proficiency (receiving a benchmark score of Close or 
Needs Support) on the grade 3 reading assessment. The probability was estimated using hierarchical logistic regression models 
with a binary outcome (demonstrating versus not demonstrating proficiency). The analysis controlled for the influences of student 
demographic characteristics and education experiences on reading proficiency. 


Quartile. A group representing 25 percent of the observed data. Quartiles are used to calculate predicted probabilities for con- 
tinuous variables. The zeroth quartile represents the sample’s minimum value, the first quartile represents the observed midpoint 
between the minimum value and the median (or the middle value in the dataset), the second quartile represents the median, the 
third quartile represents the observed midpoint between the median and the maximum value, and the fourth quartile represents 
the sample’s maximum value. 


Notes 


1. The sample used in this study showed that, contrary to policy, a small number of students were very young at kindergarten entry (between four years, 
two months old and four years, five months old). The study team conducted a sensitivity analysis that removed extreme age values and the results did 
not change substantially (see appendix D). 


2. This report uses Chamorro and Carolinian to align with the terms used by the CNMI Public School System. Some community members in CNMI, how- 
ever, may prefer CHamoru and Refaluwasch respectively. 


3. Only families that completed an application for free or reduced-price lunch benefits and received the benefit are marked as such in the data. There 
might be families that were eligible but that did not complete the application and therefore did not receive the benefit. 
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Box 2. Data sources, sample, and methods 


Data sources. The Commonwealth of the Northern Mariana Islands (CNMI) Public School System provided the study team with 
administrative records for each of the 2011/12—2018/19 school years. These records included longitudinal data from five cohorts 
of students who enrolled in grade 3 in a CNMI public school from 2014/15 to 2018/19. Student data included gender, free or 
reduced-price lunch receipt, birthdate, ethnicity, mobility, school attended, and performance on the ACT Aspire grade 3 reading 
assessment. Head Start, which is under the auspices of the CNMI Public School System, provided administrative records on Head 
Start enrollment for the 2009/10-—2015/16 school years. 


Sample. The final analytic sample for the study included 2,095 students for whom the study team received information on perfor- 
mance on the ACT Aspire grade 3 reading assessment. The sample reflects 99.1 percent of the target population of 2,115 students 
who enrolled in kindergarten in a CNMI public school from 2011/12 to 2015/16 and who enrolled in grade 3 in a CNMI public school 
from 2014/15 to 2018/19. 


Methodology. The study team used hierarchical logistic regression models to estimate the associations between the outcomes 
of interest and student demographic characteristics and education experiences. The outcome of interest was whether the student 
demonstrated proficiency (received a benchmark score of Ready or Exceeding) on the grade 3 reading assessment. Predicted 
probabilities were then calculated from the results of the hierarchical logistic regression models for demographic characteris- 
tics and education experiences that had statistically significant associations with the outcome variable. Associations for which 
the p-value associated with the coefficient was less than .05 were considered significant. Differences in predicted probabilities 
between categories that were less than 10 percentage points were classified as minor, and differences that were 10 percentage 
points or greater were classified as major. 

More information on the data sources, sample, and methodology is in appendix B. Full results of the analyses are in appendix 
C, and supplemental analyses are in appendix D. 


Findings 
This section describes the findings related to the research question. 


Female students had a higher probability of demonstrating reading proficiency in grade 3 than male 
students 


Female students were statistically significantly more likely than male students to score at the Ready or Exceed- 
ing reading proficiency benchmark in grade 3, after other student demographic characteristics and education 
experiences were controlled for (figure 1; see also tables C2 and C3 in appendix C). The predicted probability of 
demonstrating reading proficiency in grade 3 was 18 percent for female students compared with 14 percent for 
male students. The difference between those probabilities was minor. 


Students who received free or reduced-price lunch had a lower probability of demonstrating reading 
proficiency in grade 3 than students who did not receive this benefit 


Students who received free or reduced-price lunch were less likely to demonstrate reading proficiency in grade 3 
than students who did not receive this benefit, after other student demographic characteristics and education 
experiences were controlled for (see table C2 in appendix C). The probability of demonstrating reading proficien- 
cy in grade 3 was 13 percent for students who received free or reduced-price lunch compared with 19 percent for 
students who did not receive free or reduced-price lunch (see table C3). The difference between those probabili- 
ties was minor. 
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Figure 1. Female students had a higher probability of demonstrating reading proficiency in grade 3 than male 
students 


Predicted probability of demonstrating reading proficiency in grade 3 (percent) 
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Note: n = 2,095. Results are predicted probabilities derived from hierarchical logistic regression analyses and using multiple imputation of missing data. 
Gender was a statistically significant predictor (p < .01) of grade 3 reading proficiency (see table C2 in appendix C). Post-hoc comparisons indicated that 
female students had a statistically significantly higher predicted probability of demonstrating reading proficiency in grade 3 than male students (p < .01; 
see table C3). 


Source: Authors’ analysis based on 2011-19 administrative data files provided by the Commonwealth of the Northern Mariana Islands Public School System. 


Older students had a higher probability of demonstrating reading proficiency in grade 3 than younger 
students 


Students who were older at kindergarten entry were more likely to demonstrate reading proficiency in grade 3 
than students who were younger at kindergarten entry, after other student demographic characteristics and edu- 
cation experiences were controlled for (see table C2 in appendix C). The predicted probability of demonstrating 
reading proficiency in grade 3 was 10 percent for the youngest students at kindergarten entry (4 years, 2 months, 
or 50 months old) compared with 14 percent for students who entered kindergarten at age 5 years, 1 month (61 
months, 25th percentile); 16 percent for students who entered at age 5 years, 5 months (65 months; median); 
18 percent for students who entered at age 5 years, 8 months (68 months; 75th percentile); and 20 percent for 
the oldest students at kindergarten entry (6 years, or 72 months old; figure 2; see also table C3). The differences 
in probability between each age at kindergarten entry quartile were minor, except the difference between the 
youngest and oldest students at kindergarten entry. 


Filipino students had a higher probability of demonstrating reading proficiency in grade 3 than 
students of other ethnicities 


The predicted probability of demonstrating reading proficiency in grade 3 was 23 percent for Filipino students 
compared with 12 percent for Chamorro or Carolinian students and 14 percent for students who reported another 
ethnicity, after other student demographic characteristics and education experiences were controlled for (figure 
3; see also tables C2 and C3 in appendix C). The difference in probability between Filipino students and Chamorro 
or Carolinian students was major. 


Head Start enrollment was not associated with grade 3 reading proficiency 
There was no statistically significant difference in grade 3 reading proficiency between students who had enrolled 
in Head Start and students who had not, after other student demographic characteristics and education experi- 


ences were controlled for (see table C2 in appendix C). 
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Figure 2. Students who were older at kindergarten entry had a higher probability of demonstrating reading 
proficiency in grade 3 than students who were younger at kindergarten entry 


Predicted probability of demonstrating reading proficiency in grade 3 (percent) 
100 


O 
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Note: n = 2,095. Results are predicted probabilities derived from hierarchical logistic regression analyses and using multiple imputation of missing data. 
Age at kindergarten entry was a statistically significant (p < .05) predictor of grade 3 reading proficiency (see tables C2 and C3 in appendix C). The age 
values selected correspond to quartiles for the sample (see box 1 for additional details on quartiles). 


Source: Authors’ analysis based on 2011-19 administrative data files provided by the Commonwealth of the Northern Mariana Islands Public School System. 


Figure 3. Filipino students had a higher probability of demonstrating reading proficiency in grade 3 than 
students of other ethnicities 


Predicted probability of demonstrating reading proficiency in grade 3 (percent) 
100 


Filipino Chamorro or Carolinian Other 


Note: n = 2,095. Results are predicted probabilities derived from hierarchical logistic regression analyses and using multiple imputation of missing data. 
Filipino ethnicity was a statistically significant predictor (p < .001) of grade 3 reading proficiency (see table C2 in appendix C). Post-hoc comparisons indi- 
cated that Filipino students had a statistically significantly higher predicted probability of demonstrating reading proficiency in grade 3 than Chamorro 
or Carolinian students or students of other ethnicities (p < .001; see table C3). There was no statistically significant difference between Chamorro or 
Carolinian students and students of other ethnicities. 


Source: Authors’ analysis based on 2011-19 administrative data files provided by the Commonwealth of the Northern Mariana Islands Public School System. 


Students who attended the same school from kindergarten to grade 3 had a higher probability of 
demonstrating reading proficiency in grade 3 than students who changed schools 


The probability of demonstrating reading proficiency in grade 3 was 18 percent for students who attended the 
same school from kindergarten to grade 3 compared with 14 percent for students who changed schools during 
this period, after other student demographic characteristics and education experiences were controlled for 
(figure 4; see also tables C2 and C3 in appendix C). The difference between those probabilities was minor. About 
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Figure 4. Students who attended the same school from kindergarten to grade 3 had a higher probability of 
demonstrating reading proficiency in grade 3 than students who changed schools 


Predicted probability of demonstrating reading proficiency in grade 3 (percent) 
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Note: n = 2,095. Results are predicted probabilities derived from hierarchical logistic regression analyses and using multiple imputation of missing data. 
Mobility was a statistically significant predictor (p < .05) of grade 3 reading proficiency (see table C2 in appendix C). Post-hoc comparisons indicated 
that students who attended one school between kindergarten and grade 3 had a statistically significantly higher predicted probability of demonstrating 
reading proficiency in grade 3 than students who attended two or more schools from kindergarten to grade 3 (p < .05; see table C3). 


Source: Authors’ analysis based on 2011-19 administrative data files provided by the Commonwealth of the Northern Mariana Islands Public School System. 


67 percent of students in the sample attended the same school from kindergarten to grade 3 (see table C1), which 
is somewhat higher than the national average (almost 56 percent; see Burkam et al., 2009). 


Limitations 
Several issues limit the interpretation of findings from this study. 


First, as with other analyses using observational data and nonexperimental analytic methods, the results cannot 
be interpreted as causal. That is, it cannot be said that the study predictors lead to the observed reading out- 
comes. Other explanations could exist. 


Second, limited data prevented the study team from analyzing other factors that could influence grade 3 reading 
proficiency. Complete data were not available on teacher education and credentials, instructional quality, or 
student absenteeism, which the study team hypothesized, based on a literature review, might be associated with 
students’ grade 3 reading performance. Because data were not available, the study team could not include those 
measures in the analyses. Although some information was available on the quality of classroom instruction, it 
was not possible to directly link student records to classroom teacher records. Doing so might have provided 
useful insight into student reading performance. It was also impossible to explore the wide range of differences in 
proficiency rates across schools, which might have provided further insight into the needs of specific schools and 
the strategies that could address them. Similarly, the Head Start measure provided information only on student 
enrollment and not on participation in the program or use of Head Start services, so it was not possible to explore 
these effects on grade 3 reading proficiency. Additionally, data were unavailable on other early childhood experi- 
ences available to CNMI children. 


Limited data also meant that the study team did not explore many other factors that research has suggested 
could be associated with grade 3 reading proficiency. These include student health records, indicators of adverse 


childhood experiences, students’ participation in out-of-school activities, parents’ employment and education 
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status, school curricula, and other measures of cognitive, emotional, or social development. The extent to which 
the CNMI Public School System systematically collects these data is unclear. It is possible that these factors, rather 
than the factors examined, have a larger influence on differences in reading proficiency. Understanding these 
factors would require a separate study. 


Third, few of the examined factors predicted meaningful or large differences in reading performance among stu- 
dents. Given the minor differences found among the factors analyzed in this study, it is difficult to derive specific 
policy implications without further analysis. Although the study found a major difference in reading proficiency 
between Filipino students and their peers, identifying policy changes or practices from this finding will require 
further study of the underlying behaviors or supports to which these students have access. This is discussed 
further in the implications section. 


Implications 


This study found that some student demographic characteristics and education experiences had a positive, if 
small, association with reading outcomes among CNMI grade 3 students but that those factors do not fully explain 
students’ reading proficiency. The study findings have three main implications. 


Examine additional education environment factors that might influence student reading proficiency 


CNMI Public School System stakeholders might consider seeking to better understand how other education envi- 
ronment factors are related to student reading proficiency. Though the study was unable to explain the variance 
in grade 3 reading performance across schools, exploring relationships between other education environment 
factors and student performance might help explain that variance and might provide CNMI Public School System 
greater insight into the school characteristics or experiences that are stronger drivers of student reading profi- 
ciency. These other factors include teacher education, teacher academic skills, and teacher years of experience. 
Prior research suggests that exposure to high-quality teaching can lead to better student performance (Atlay 
et al., 2019; Desimone & Long, 2010; Hanushek & Rivkin, 2006). One study that examined the association between 
teaching experience and teaching effectiveness found that students of teachers with more experience (those 
teaching for seven or more years) had higher performance on standardized assessments (Kini & Podolsky, 2016). 
Other characteristics of the education environment in elementary schools, such as low turnover and positive 
school climate, have been found to maintain student gains, especially for students who attend preschool (Ansari 
& Pianta, 2018; Unterman & Weiland, 2019). The CNMI Public School System is building a longitudinal data system 
that will better link teacher and student data and can facilitate future research on these questions. 


Collect additional information to better understand the academic performance of different student groups 


CNMI Public School System stakeholders might benefit from conducting additional research that focuses on 
understanding the differences in academic outcomes among their students. The largest difference in grade 3 
reading performance that the study found was between Filipino students and their non-Filipino peers, while 
other differences, such as those between students who changed schools and students who did not change 
schools, were minor, though statistically significant. Supplemental analyses showed that Filipino students had a 
higher probability of having enrolled in Head Start than their peers. However, having enrolled in Head Start did 
not explain Filipino students’ higher probability of demonstrating reading proficiency in grade 3 (see table D5 in 
appendix D). The explanation behind this finding warrants future study. Research suggests that, rather than focus- 
ing on uncontrollable categories such as ethnicity or free or reduced-price lunch receipt, future studies could 
observe potential behaviors or processes that schools might have more influence over, such as providing parent 
education programs on the importance of supporting reading skills at home (Raag et al., 2011). More detailed 
qualitative data (collected via surveys, interviews, or focus groups) might help elucidate the underlying behaviors 
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or supports that Filipino students and their families have that could explain their performance. This information 
could then be used to target other students with additional resources to raise their overall reading achievement. 


Track the academic outcomes of Head Start enrollees and their peers with other or no early childhood 
education experiences at additional time points, including kindergarten entry, middle school, and 
high school 


The CNMI Public School System might consider studying the effects of Head Start enrollment at additional points 
during students’ education. One possible reason for the lack of observable grade 3 differences in this study is 
that students who had not enrolled in Head Start caught up with their peers during the years from kindergarten 
to grade 3. Another possible reason is that Head Start attendance or active participation—rather than enroll- 
ment, the factor examined in this study—drives differences in academic outcomes. These possibilities, or other 
unobserved student demographic characteristics or education experiences, require further exploration (Meloy 
et al., 2019). Other studies have found that Head Start enrollment might not correlate with standardized reading 
scores but might correlate with other later academic outcomes, such as middle school math test scores (Phillips 
et al., 2016) and high school graduation (Deming, 2009). CNMI stakeholders could track the performance of Head 
Start enrollees and students with other or no early childhood education opportunities to determine whether early 
childhood effects are found in kindergarten, middle school, and high school. This would enable future studies to 
examine both short- and long-term measures of academic outcomes (in other subjects besides reading and in 
other grades besides grade 3) and nonacademic outcomes. Prior research indicates that Head Start and other 
center-based early childhood education programs have similar impacts on student outcomes (Feller et al., 2013; 
Meloy et al., 2019). Collecting information on the other types of early childhood education programs in which 
children enrolled prior to kindergarten might allow for a more robust understanding of the reasons behind the 
varying performances between student groups. It might also provide insights into both academic outcome trends 
across these experiences and into how stakeholders could support students more effectively. 
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